Bounding the k-family-wise error rate using resampling methods

The multiple hypothesis testing (MHT) problem has long been tackled by controlling the family-wise error rate (FWER), which is the probability that any of the hypotheses tested is unjustly rejected. The best known method to achieve FWER control is the Bonferroni correction, but more powerful techniques such as step-up and step-down methods exist. A particular challenge to be dealt with in MHT problems is the unknown dependency structure between the tests. The above-mentioned approaches make worst-case assumptions in this regard, which makes them extremely conservative in practical situations, where positive dependencies between the tests are abundant. In this paper we consider randomisation strategies to overcome this problem, and provide a rigorous statistical analysis of the finite-sample behaviour. Furthermore, we extend our results to an approach to control the k-FWER as introduced by [7]. Another result is a uniform bound on the k-FWER, uniform over a specified set of values of k, which additionally allows to control the false discovery proportion (FDP, see e.g. [7]). Our methods are essentially assumption free, and by effectively taking into account dependencies between the tests their strong power is ensured in all situations.